Repository navigation
CLI: view alias, sweep commands, recover fix - #1223
Merged
Merged
Conversation
nemar dataset view now resolves to the same status command.
Description text and AGENTS.md CLI table mention the alias.
Tested: bun test test/cli.test.ts -t 'Dataset Status' (6 pass); confirmed the 3 new assertions fail when .alias("view") is reverted.
nemar admin recording-stats-sweep and nemar admin signal-defaults-sweep call the existing backend admin routes, mirroring hedSweepCommand's batch-loop/--reset/--verbose/--json shape. Client calls added to src/lib/api/admin.ts; both commands added to the AGENTS.md CLI table. Tested: bun test test/recording-signal-sweeps-cli.test.ts (19 pass), real subprocess CLI against a local capture server (no mocks); confirmed 16/19 assertions fail when the two addCommand() registrations are removed.
process.env.X = previous poisons X to the literal string "undefined" when it was never set, instead of deleting it; root bun test runs test/ and backend/test/ in one process, so this leaked into later files. Fixed the unguarded restores in test/maintenance-client.test.ts and test/curl-stream-copy.unit.test.ts to match test/publish-progress.test.ts's guarded pattern, and trimmed search-color's now-stale workaround comment (the TEST_API_URL override there is still needed, just no longer as a bug workaround). Tested: bun test on all three files (17 pass); reproduced the poisoning with a throwaway probe (checkHealth() failed with "Could not connect to undefined" when the guard was reverted) and confirmed the guarded restore fixes it.
scripts/recover-datasets.json was never shipped (package.json files omitted it) and defaultRecoverDatasetsPath() hardcoded a source-tree-only depth (two levels up from src/commands/), which lands outside the installed package entirely from the single-file dist/index.js bundle. Now ships the file via files and tries the source-tree path first, falling back to the bundled (one level up) path. Tested: bun test test/recover-datasets-packaging.test.ts (5 pass) -- asserts the real bun pm pack tarball includes the file, and runs the real nemar admin recover --all --json from both bun run src/index.ts and a real bun build dist/index.js bundle placed at the npm-installed layout. Confirmed 3/5 assertions fail when the fix is reverted (the packed-file-list check and the dist-bundle run; the source-mode run was never broken, as expected).
The git-state reconcile (listTrackedPaths / computeAddTargets) lived entirely inside the s3_upload-incomplete gate, so once a prior run stamped s3_upload complete, a data file added afterward was silently never picked up again on a resumed upload. Moved the cheap reconcile above the gate so it always runs; when it finds real work despite the stale completion stamp, it clears s3_upload so the existing gate naturally reopens. The expensive upload itself stays gated exactly as before. Tested: bun test test/upload-s3-resume.unit.test.ts (2 pass) -- drives the real uploadDataToS3 entry point against a real git-annex repo on disk; confirms a new file gets git-annex tracked and the stale s3_upload stamp clears even though a real local credentials server always errors (proving the reconcile path, not a full S3 round-trip, which needs live AWS creds and belongs to the e2e suite). Confirmed the new assertions fail (result stays "ok", file stays untracked) when the fix is reverted. Sibling suites (annex-add-targets, upload-progress, upload-steps) still pass: 97 tests.
Critical review finding on PR #1223: reopening s3_upload for new content (issue #1070's fix) left dataset_save/github_push stamped complete from the prior run, so a resumed upload could re-upload a new file to S3 but never commit or push its git-annex pointer -- and printUploadSuccess would still clear progress and report success. clearStepCompleted now also clears dataset_save/github_push whenever the reconcile reopens s3_upload, and the two/three writeUploadProgress calls collapse into one combined write (review suggestion 4, same fix). Tested: bun test test/upload-s3-resume.unit.test.ts (3 pass) -- the new test pre-stamps all four steps complete, adds a file, drives the real uploadDataToS3 then the real saveDatasetStep, and asserts dataset_save/ github_push are reopened and the new file actually gets committed. Confirmed it fails (dataset_save stays true) when the fix is reverted.
Important review finding on PR #1223: the same packaging bug the recover fix addressed for scripts/recover-datasets.json also applies to scripts/withdrawn-datasets.json (defaultWithdrawnDatasetsPath) -- confirmed missing from the tarball with bun pm pack --dry-run. Same fix as recover: added to package.json's files, and defaultWithdrawnDatasetsPath tries the source-tree depth first, falling back to the bundled depth. Tested: bun test test/recover-datasets-packaging.test.ts (10 pass, extends the existing recover coverage) -- asserts the real bun pm pack tarball includes the file, and runs the real nemar admin withdraw --all --json (via a local stub answering the withdraw endpoint, since even withdraw's dry run calls the backend once per target) from both bun run src/index.ts and a real bun build dist/index.js bundle. Confirmed 3/10 fail when the fix is reverted.
Suggestion 5 on PR #1223 review: hed-sweep, recording-stats-sweep, and signal-defaults-sweep duplicated the same do-while batch loop, progress guard, GitHub-pacing sleep, and try/catch three ways. runSweepBatchLoop now owns that shared control flow; each command's totals accumulation and verbose/summary text stay inline via a callback, so no command's actual output changes. Also closes a pre-existing gap: hed-sweep had no CLI-level test asserting its exact output before this. test/hed-sweep-cli.test.ts adds that coverage and doubles as the regression guard for the extraction. Tested: bun test test/hed-sweep-cli.test.ts (10 pass) and test/recording-signal-sweeps-cli.test.ts (19 pass, unchanged assertions on exact stdout still hold). Verified byte-identical --help output for all three commands before/after, and confirmed test/hed-sweep-cli.test.ts passes identically against the pre-extraction (triplicated) code, proving the refactor is behavior-preserving rather than merely matching new code. Found and deliberately left alone (out of scope, pre-existing, filed as a follow-up candidate): the root CLI's global --verbose option (src/index.ts:54) shadows every subcommand's own local --verbose (including these three, plus admin.ts:3372 and :4750), so options.verbose is always undefined in these action handlers and --verbose is currently a silent no-op on all of them.
📦 Package PreviewYour package is ready to test! bun install -g nemar-cli@PR1223Version: 🤖 Updated on each push. |
unit-pure failed on #1223 because test/api-export-surface.unit.test.ts pins lib/api/admin's runtime exports; the two sweep commands added recordingStatsSweep, recordingStatsSweepReset, signalDefaultsSweep and signalDefaultsSweepReset. Added to the admin pin and to POST_SPLIT_ADDITIONS with their issue reference. Tested: bun test test/api-export-surface.unit.test.ts (10 pass).
This was referenced Sep 3, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Five small, independently reviewable CLI-side fixes, one commit each.
nemar dataset viewis now an alias fornemar dataset status, registered via Commander's.alias()and mentioned in the command description and the AGENTS.md CLI table.nemar admin recording-stats-sweepandnemar admin signal-defaults-sweepare new commands mirroringhedSweepCommand's batch-loop/--reset/--verbose/--json shape, calling the existingPOST /datasets/recording-stats-sweepandPOST /datasets/signal-defaults-sweepbackend routes via new client functions insrc/lib/api/admin.ts.Three
afterAllhooks restoredprocess.env.Xwith an unguarded assignment (process.env.X = previous), which coercesundefinedinto the literal string"undefined"instead of deleting the key; since rootbun testrunstest/andbackend/test/in one process, this leaked into later test files. Fixedtest/maintenance-client.test.tsandtest/curl-stream-copy.unit.test.tsto matchtest/publish-progress.test.ts's existing guarded pattern, and trimmedtest/search-color.unit.test.ts's now-stale workaround comment.nemar admin recoverwas broken in every published npm build:scripts/recover-datasets.jsonwas never shipped (missing from package.json'sfiles), and the default path resolution hardcoded a source-checkout-only depth that lands outside the installed package from the single-filedist/index.jsbundle. Now ships the file and tries the source-tree path first, falling back to the bundled depth.uploadDataToS3's git-state reconcile (listTrackedPaths/computeAddTargets) lived entirely inside thes3_upload-incomplete gate, so a data file added to a dataset after a prior run completed was silently never picked up on a resumed upload. The reconcile now runs unconditionally (cheap, no content access) and reopens thes3_uploadgate when it finds real work; the expensive upload itself stays gated exactly as before.Closes #1212
Closes #1194
Closes #1175
Closes #1049
Closes #1070
Test plan
bun test test/cli.test.ts -t "Dataset Status"— 6 pass (alias resolves identically tostatus, including--helptext anddataset --help'sstatus|viewlisting).bun test test/recording-signal-sweeps-cli.test.ts— 19 pass, real subprocess CLI against a local capture server (no mocks); confirmed 16/19 assertions fail when the two newaddCommand()registrations are removed.bun test test/maintenance-client.test.ts test/curl-stream-copy.unit.test.ts test/search-color.unit.test.ts— 17 pass; reproduced theTEST_API_URLpoisoning with a throwaway probe (checkHealth()failed with "Could not connect to undefined" when the guard was reverted) and confirmed the guarded restore fixes it.bun test test/recover-datasets-packaging.test.ts— 5 pass; asserts the realbun pm packtarball includes the file, and runs the realnemar admin recover --all --jsonfrom bothbun run src/index.tsand a realbun builddist/index.jsbundle placed at the npm-installed layout. Confirmed 3/5 fail when the fix is reverted.bun test test/upload-s3-resume.unit.test.ts— 2 pass, real git-annex repo on disk; confirmed the new assertions fail (result stays "ok", new file stays untracked) when the fix is reverted. Sibling suites (annex-add-targets,upload-progress,upload-steps,recover-datasets) still pass: 159 tests total across all touched + sibling files.bun run lint— clean (biome check src/).bun run typecheck— clean (tsc --noEmitfor both root andbackend/).bun run build(the project's real build script,--target bun) — succeeds; verifiednemar admin recoverworks from the built bundle.bun test test/cli.test.tshas 13 pre-existing failures unrelated to this PR (auth/admin/upload/collaborator tests requiring a live backend +TEST_ADMIN_API_KEY/TEST_USER_API_KEYfromtest/.env.test, which is not present in this environment); none of them touch code this PR changes.